Skip to content

feature: brokerage order polling service - #9696

Draft
Romazes wants to merge 21 commits into
QuantConnect:masterfrom
Romazes:feature-order-polling-service
Draft

feature: brokerage order polling service#9696
Romazes wants to merge 21 commits into
QuantConnect:masterfrom
Romazes:feature-order-polling-service

Conversation

@Romazes

@Romazes Romazes commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

A reusable order polling service in Lean core, so a brokerage plugin can read the order lifecycle over HTTP when it has no stream, when its stream dies, or when the broker never replied about one order.

  • Brokerages/Services/: abstract BrokerageOrderPollingService — the loop, the watch registry, the state compare, the watch timeout and the repeated-failure warning — with the two read modes as subclasses: PerOrderIdPollingService (get-order endpoint) and AllOrdersPollingService (bulk endpoint). The plugin converts its wire model into BrokerOrderState snapshots; the shared compare reports submits, fills and closes, each exactly once.
  • BrokerageMessageQueue: the lock and queue behind BrokerageConcurrentMessageHandler<T>, split out so one handler can carry a second message type (RegisterMessageType<TMessage>). Polled snapshots queue behind an order request that holds the stream lock, so a fill can never be reported before its submit. The generic handler keeps its exact public surface — the thirteen plugins using it today do not change.
  • SeedAndStart(drainBufferedMessages, seed): the handover from a dying stream in one call, in the only safe order — process what the stream already delivered, seed one watch per open Lean order, then start — so the first sweep repeats nothing the stream reported.

The full design, the alternatives considered and the plugin survey live in Documentation/ADR/0001-brokerage-order-polling-service.md, included in this PR.

Related PR(s)

QuantConnect/Lean.Brokerages.CharlesSchwab#107 — the first adopter (draft): Charles Schwab replaces its own polling fallback with this service.

Related Issue

N/A

Motivation and Context

The same polling loop and state compare is already written three times — Public.com, Tradier and Charles Schwab — and InteractiveBrokers blocks the order thread for up to five minutes waiting for a submission event instead. One core service replaces the copies and gives every plugin the same behavior.

Requires Documentation Change

No — the design document ships in this PR (Documentation/ADR/0001-brokerage-order-polling-service.md).

How Has This Been Tested?

  • BrokerageOrderPollingServiceTests — 29 tests: submit exactly once, cumulative fill compare (never repeats, never invents a price), combo fills split by GroupOrderManager, terminal exactly once, seeded watches, watch timeout, failure warning once per outage, SeedAndStart ordering and guards.
  • BrokerageMessageQueueTests — 4 tests: two message types share one lock and one queue.
  • BrokerageConcurrentMessageHandlerTests — the existing 8 tests pass unchanged; the handler's public surface is untouched.
  • First adopter compiled and exercised against this branch in the Charles Schwab draft PR.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

Romazes added 15 commits August 18, 2026 17:26
- one core service polls order updates when the stream is missing, lost or silent
- two modes by constructor: read one watched brokerage order id, or all orders
- brokerages convert their wire model into a shared BrokerOrderState and one core diff emits the events
- grounded in a survey of eight brokerage plugins
…ge queue

- BrokerageMessageQueue owns the lock, buffer and dispatch, raised through a MessageReceived event
- BrokerageConcurrentMessageHandler<T> becomes a thin wrapper with the same public API
- add RegisterMessageType<TMessage> and a generic HandleNewMessage<TMessage>, so a second message source can queue behind the same lock instead of a separate, independent one
- add BrokerageMessageQueueTests covering the shared lock, an unmatched message type, and a registered second type
- abstract BrokerageOrderPollingService with per-order-id and all-orders modes: watch registry, state compare, watch timeout, repeated-failure warning
- plugins seed the registry before Start, so the first sweep repeats nothing the stream already reported
- ADR: seed-before-start handover and the survey of streaming plugins that fit it
- tests: 25 polling service, 4 message queue
- guard, buffered-message processing, one seed per open Lean order, Start - in the only safe order
- both callbacks optional and nullable
- add non-generic BrokerageConcurrentMessageHandler with per-type Register, reusing the generic handler by composition
- restore BrokerageConcurrentMessageHandler<T> to master and drop BrokerageMessageQueue
- service constructor takes the handler, registers ProcessOrderState and routes the states; SeedAndStart drains the handler internally
- update the ADR wiring, seed-before-start and risk sections, and note the CharlesSchwab pilot
- a plain place watches the main id; the first sweep assigns the leg ids by symbol and reports the submit through the diff
- the assignment is one shared method with the stream's OrderAccepted and releases the three minute place wait in both modes
- the pilot note lists the leg id assignment and the replace reporting as what stays in the plugin
- watch replacement watches the new id of a replace and drops the replaced one
- the diff's first state for a marked id reports the update submit
- the new id starts with no fill state; carry-over brokers seed instead
- survey of nine sibling plugins' update paths, checked line by line
- replacement watch design, the no-wait rule, and the rejected event reuse
- status entries: pilot verified live on real accounts; public.com is the second plugin on the service
- pricing text matches what shipped: public keeps its change-of-average recovery inside its mapping
- rollout item 3 trimmed to the one intentional behavior change
- protected create overloads pick the per-order-id or all-orders mode by the read callback
- the service events forward onto the brokerage events; a virtual method owns the not-acknowledged warning
- dispose covers the created service
- status entries for the wiring seam and the third plugin
- cross-zero section: leg chaining from the read and the fill-offset seed
- wiring examples moved onto the create seam
- testing section: live capture first, recorded mock data, offline checklist
- fixtures play the transaction handler: write each reported status back onto the lean order and assert in live cycle order
- rename SeedAndStart(seed) to the Start(preLoadOpenOrders) overload
- add BrokerOrderState constructors, one overload takes the message without fills
- trace the wired service with its intervals and the pre-load counts
- record public.com's live-first testing run and the offline checklist additions
- tradier reads per id and has draft pr; stale refs and tenses fixed
- the pre-loading start, the state constructors and the price-recovery note
@Romazes
Romazes force-pushed the feature-order-polling-service branch from 6a81626 to e274b06 Compare August 18, 2026 14:26
- move the service to Brokerages/Services/OrderPolling with its shapes in Models
- rename OrderNotAcknowledged to BrokerageOrderNeverNotified across the family
- the args resolve the Lean order, carry WatchDuration and print via ToString
- the default warning uses the args ToString; the message code string stays
…time

- rename BrokerOrderState to BrokerageOrderSnapshot across the service and tests
- drop the property-fill constructor, every builder goes through the constructors
- timeUtc is nullable in the constructors and defaults to DateTime.UtcNow
- shorten the snapshot xml docs
- SingleOrderPollingService and BulkOrdersPollingService, base takes the Base prefix
- callbacks are getBrokerageOrderById and getAllBrokerageOrders; watchTimeout is notificationTimeout
- both mode classes chain three constructors instead of optional parameters
- defaults read 3 * 1000 and 60 * 1000 ms; usage remarks with a wiring example on both modes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant